System, operation cell, method, product manufacturing method, and marker for locating operation position

ABSTRACT

An operation position locating system locates a position on a product where an operation is performed in a manufacturing process of the product including one or more operations to be performed. The system includes an operation completion coordinates detection unit and an operation position locating unit. The operation completion coordinates detection unit processes a plurality of images obtained by capturing from a plurality of viewpoints a range including an operation completion position where an operation is completed when the operation is completed, thereby detecting a set of operation completion coordinates representing the operation completion position. The operation position locating unit obtains a set of operation coordinates representing an operation position where the operation is to be performed on each operation, and locates the operation position corresponding to the operation completion position based on the set of operation completion coordinates and the one or more obtained sets of operation coordinates.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the technology of locating the positionof an operation performed in a manufacturing process of a product.

2. Description of the Related Art

In a manufacturing process of a product requiring a number of operationsto be performed by workers, there often occur erroneous operations by aworker forgetting an operation or failing to complete an operation. Forexample, the manufacturing process of a large electronic equipment unit,a printed circuit board unit, a communication equipment unit, etc.includes an assembly process of parts, and the assembly process includesa number of screwing operations for attaching a part to another part. Inthe assembly process, erroneous operations frequently occur fromscrewing operations.

Examples of erroneous operations are stopping a screwing operationbefore reaching a specified torque of a screw, forgetting to complete ascrewing operation after a temporary screwing operation, forgetting toplace a screw at a position where the screwing operation is to beperformed, etc. Since there are various types of erroneous operations,there are countermeasures to be taken from various viewpoints to reducethe erroneous operations

For example, to tighten a screw with a specified torque, it is effectiveto use a device called a torque driver designed for starting idlerunning when an applied torque is detected to reach a specified torque.The device is used in an assembly process by a worker or in an assemblyprocess by a robot. The patent document 1 describes a thread fasteningdevice (namely, screwing device) for an assembly process by a robot. Thethread fastening device is fixed to the hand of the robot, and detectsthe coordinates of a screw hole by recognizing an image. When a robotcontroller corrects the position of the robot based on the coordinates,the thread fastening device drives a nut runner, and tightens the screwwhile monitoring the applied torque and the driving time.

To reduce erroneous operations of forgetting an operation to beperformed, there have conventionally been operation tools such as ascrewing counter, an electronic shelf, etc. By using these operationtools, it is possible to compare the number of operations to beperformed with the number of operations practically performed and detectforgetting an operation. However, it is not possible only by comparingthe numbers to concretely determine which operation has been forgotten.Therefore, it is necessary to visually confirm the point where thescrewing operation has been forgotten when inconsistency is detected asa result of the comparing the numbers. At present, there are still anumber of erroneous operations even using these operation tools. As aresult, there frequently occurs defective manufacture, which requiresinspection and repair within a manufacturing process.

Thus, in a manufacturing process to be performed by a number ofoperations by workers, forgetting operations is a serious problem. Thepatent document 2 describes a work quality control system for solvingthe above-mentioned problem. In this work quality control system, aplurality of receivers receive a signal issued by an ultrasonictransmitter attached to the tip of a nut runner. Based on the differencein propagating time of a signal and the positional relation betweenreceivers, the position of the transmitter is located. Since theplurality of receivers are set such that they can keep a predeterminedrelative positions from a workpiece to be assembled, the position of atightened screw is detected from the position of the transmitter basedon the relationship between the static relative positions. The locationinformation indicating the thus detected position of the tightened screwand the fastening information indicating a result of the comparisonbetween the actually applied torque and the specified torque are managedin combination.

[Patent Document 1] Japanese Published Patent Application No. H6-23631

[Patent Document 2] Japanese Published Patent Application No.2004-258855

However, there are the following two problems with the system describedin the patent document 2.

The first problem is that it is necessary to attach a transmitter ofultrasonic signal or radio signal to a tool. Such necessity cause aproblem because the transmitter requires more power, the wiring forsupply of power to the transmitter can be the obstacle to an operation,and/or the transmitter can not be attached to some types of existingtools. It is important for an invention for reducing forgetting anoperation in a manufacturing process to be easily implemented in theexisting manufacturing process.

The second problem is an assumption that it is predetermined that thepositional relation between a workpiece and receivers is staticallydetermined. In the product manufacturing process, there are processsteps in which the position and the direction of a product inmanufacturing is not predetermined. In some process steps, the positionand the direction of a product are uncertain until a practical operationis performed due to various reasons including the convenience of aworker, etc. In the case in which one process includes a plurality ofoperations, the process can be performed with a different position ordirection depending on each operation.

SUMMARY OF THE INVENTION

The present invention aims at providing an operation position locatingsystem locating a position on a product where an operation is performed,and especially an operation position locating system provided with atleast one of the feature that no signal transmitter is required and thefeature that the system can be applied to a case in which the positionor direction of a product can be dynamically changed.

It is also an objective of the present invention to provide an operationcell using the operation position locating system, a method performed inthe operation position locating system in a manufacturing process of aproduct, and a program used to direct a computer to perform the method.A marker preferably used in the operation position locating system mayalso be provided in an embodiment.

An operation position locating system in an embodiment according to thefirst aspect of the present invention locates a position on a productwhere an operation is performed in a manufacturing process of theproduct including one or more operations to be performed using a tool.The operation position locating system comprises an operation completioncoordinates detection unit and an operation position locating unit.

The operation completion coordinates detection unit processes aplurality of images obtained by capturing from a plurality of viewpointsa range including an operation completion position as a position onwhich an operation is completed when the operation is completed, therebydetecting a set of operation completion coordinates representing theoperation completion position.

The operation position locating unit obtains a set of operationcoordinates representing an operation position as a position on whichthe operation is to be performed on each of the one or more operations,and locates the operation position corresponding to the operationcompletion position based on the set of operation completion coordinatesand the one or more obtained sets of operation coordinates.

According to the operation position locating system in the embodiment ofthe first aspect, the operation completion position is detected by imageprocessing, and therefore it is not necessary to provide a signaltransmitter to the tool.

An operation position locating system in an embodiment according to thesecond aspect of the present invention locates a position on a productwhere an operation is performed in a manufacturing process of theproduct including one or more operations to be performed using a tool.The operation position locating system comprises an operation completioncoordinates detection unit, a parameter determination unit, and anoperation position locating unit.

The operation completion coordinates detection unit detects a set ofoperation completion coordinates in which an operation completionposition as a position where the operation is completed is representedby a first coordinate system.

The parameter determination unit detects a position and a directionrepresented by the first coordinate system of the product or an objecthaving a predetermined positional relation with the product, therebydetermines a parameter value required to coordinate transformationbetween the first coordinate system and a second coordinate system basedon the product.

The operation position locating unit obtains a set of operationcoordinates in which an operation position as a position where theoperation is to be performed is represented by the second coordinatesystem on each of the one or more operations, transforms coordinatesusing the parameter to represent the operation completion position andthe operation position by a same coordinates system, and locates theoperation position corresponding to the operation completion position.

According to the operation position locating system in the embodiment ofthe second aspect, the operation position corresponding to the operationcompletion position can be located even if the position and thedirection of the product being manufactured can be dynamically changedin the manufacturing process of the product.

The present invention can also be embodied as an operation cell usingthe operation position locating system in either embodiment according tothe first or second aspect of the present invention, a method performedin the operation position locating system in a manufacturing process ofa product in either embodiment according to the first or second aspectof the present invention, or a program used to direct a computer toperform the method.

According to the above-mentioned operation position locating systems,the position on the product where an operation has been performed can belocated.

Therefore, by using the result of location by the operation positionlocating system, an operation actually performed can be distinguishedfrom an operation not yet performed in the operations to be performed.Accordingly, it can be automatically checked whether or not there is amissed operation in the manufacturing process. Thus, the number ofman-hours for an operation instruction steps and testing process stepsto be performed after the manufacturing process can be reduced. Inaddition, since the frequency of occurrences of defective productscaused by forgetting an operation can be reduced, the manufacturingyield can also be enhanced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the function showing the configurationaccording to a mode for embodying the operation position locating systemof the present invention;

FIG. 2 shows the configuration of the workpiece assembly line includingthe operation position locating system;

FIG. 3 is a schematic chart showing the flow of the process in anoperation cell;

FIG. 4 is a perspective view showing the configuration of the operationcell according to a mode for embodying the present invention;

FIG. 5 shows the connection of the electronic equipment relating to thelocation of an operation position;

FIG. 6 is a schematic and perspective view showing the details obtainedby enlarging a part of FIG. 4;

FIG. 7 is a flowchart of the flow of the process according to a mode forembodying the present invention;

FIG. 8 is a schematic chart showing the positional relation of threedetected markers;

FIG. 9 shows an example of an arrangement pattern of markers;

FIG. 10 is a schematic chart for explanation of the method ofcalculating the coordinates of the tip of the driver from thecoordinates of markers;

FIG. 11 shows an example of a display screen of an operation monitor;

FIG. 12 shows an example of a marker to be attached to an operationtable;

FIG. 13A shows an example of a marker to be attached to a driver; and

FIG. 13B shows an example of a marker to be attached to a driver.

DESCRIPTION OF THE EMBODIMENTS

The modes for embodying the present invention are described below indetail with reference to the attached drawings. Embodiments applied tothe screwing operation in assembling a printed circuit board unit ismainly described below, but it is obvious that the embodiments of thepresent invention are not limited with respect to the type of product oroperation to which the embodiments of the present invention is applied.

FIG. 1 is a block diagram of the function according to a mode forembodying the present invention of the operation position locatingsystem. In the present embodiment, in the manufacturing process of aproduct including one or more operations to be performed using a tool,an operation practically performed is distinguished in the operations tobe performed by locating the position on the product where the operationhas been performed.

When one operation is completed, a completion signal output unit 1outputs a completion signal, and an operation completion coordinatesdetection unit 2 receives the completion signal. In the followingdescriptions, the position where an operation is completed is called an“operation completion position”, and the coordinates representing theoperation completion position by the fixed and global first coordinatesystem are called an “operation completion coordinates”. In thisembodiment, the operation completion position is represented by atriplet of operation completion coordinates because the first coordinatesystem is a three-dimensional coordinate system.

In more detail, the operation completion position refers to a positionof a target of an operation when the operation is completed. Forexample, the target of a screwing operation is a screw, and the positionof the screw when the operation is completed is the operation completionposition. To be more correct, the position of a specific point of ascrew predetermined as indicating the representative position of thescrew is the operation completion position. When the screwing operationis performed, the tip of the driver contacts the screw, and theoperation completion position is also the position where the toolcontacts the target of the operation when the operation is completed.

Since the time required to propagate a completion signal is so short asto be ignored without any problem, the position where the operation isbeing performed when the operation completion coordinates detection unit2 receives the completion signal can be regarded as the operationcompletion position. The operation completion coordinates detection unit2 detects an operation completion coordinates when it receives thecompletion signal.

In the present mode for embodying the present invention, the position ofa marker attached to a tool is detected in the image recognizingprocess, and the operation completion coordinates are calculated fromthe position of the marker, thereby detecting the operation completioncoordinates. A marker can be made of resin or rubber, and can be easilyattached to a tool with an adhesive. In the image recognizing process, aplurality of images captured from a plurality of different viewpointsare used. The plurality of images are images which capture the rangeincluding the operation completion position. Typically, two images areused, and the operation completion coordinates are calculated on thebasis of the principle of stereo vision.

The position where each operation should be performed is predetermined.The position is hereinafter referred to as an “operation position”. Inmore detail, the coordinates of the operation position represented bythe local second coordinate system based on a product is predetermineddepending on the product to be manufactured. The coordinates arehereinafter called “operation coordinates”. In this embodiment, theoperation position is represented by a triplet of operation coordinatesbecause the second coordinate system is also a three-dimensionalcoordinate system.

The operation coordinates are determined when a product is designed, andindicate an ideal position without an error, that is, a nominalposition. An actual operation may be accompanied by an error, and may beperformed at a position a little displaced from an operation position.

An operation coordinates storage unit 3 stores a set of operationcoordinates of each operation. Specifically, in this embodiment, eachtriplet of operation coordinates for each operation is stored in theoperation coordinates storage unit 3. The operation coordinates can alsobe stored in the operation coordinates storage unit 3 in advance, andcan be externally provided as necessary and stored in the operationcoordinates storage unit 3.

A parameter determination unit 4 detects the position and directionrepresented by the first coordinate system of a product or an objectsuch as an operation table etc. having a predetermined positionalrelation with the product, and determines the value of a parameterrequired for coordinate transformation between the first coordinatesystem and the second coordinate system. The coordinate transformationcan be a transformation from the coordinates in the first coordinatesystem to the coordinates in the second coordinate system, or can be theinverse transformation, namely, a transformation from the secondcoordinate system to the first coordinate system. However, the former ismore effective and preferable as described later.

An operation position locating unit 5 performs the coordinatetransformation for representing both operation completion position andoperation position in the same coordinates system using a parameterdetermined by the parameter determination unit 4, and locates (namely,identifies) the operation position corresponding to the operationcompletion position.

For example, in the mode for embodying the present invention fortransforming the coordinates in the first coordinate system to thecoordinates in the second coordinate system, the operation completioncoordinates detected by the operation completion coordinates detectionunit 2 are transformed to the coordinates in the second coordinatesystem by the operation position locating unit 5. The operation positionlocating unit 5 reads the operation coordinates represented in thesecond coordinate system from the operation coordinates storage unit 3,and locates the operation position corresponding to the operationcompletion position based on the read operation coordinates and thetransformation result.

In a case where the operation position locating unit 5 transforms thecoordinates in the second coordinate system to the coordinates in thefirst coordinate system, it is obvious that the operation positionlocating unit 5 can locates the operation position corresponding to theoperation completion position in the similar method.

With the above-mentioned configuration, each time an operation iscompleted, the parameter indicating the relationship between the firstcoordinate system and the second coordinate system is dynamicallycalculated. Thus, the operation position and the operation completionposition can be represented and compared in the same coordinates systemeven if the position and the direction of a product can be dynamicallychanged when an actual operation is performed. Therefore, the operationposition corresponding to a completed operation can be located. As aresult, the completed operation in the operations to be performed can bediscriminated.

FIG. 2 shows the configuration of the workpiece assembly line includingthe operation position locating system. Each block shown in FIG. 2 is afunctional block.

The product to be manufactured is assembled by a workpiece assembly line100. The workpiece assembly line 100 includes a plurality of operationcells 101 through 104. One process step of the assembly process of aproduct corresponds to one operation cell.

The product being assembled is hereinafter referred to as a “workpiece”.A workpiece is fed to the operation cells 101, 102, 103, and 104sequentially. Each operation cell is preassigned one or more operationsto be performed on a workpiece. That is, one process step includes oneor more predetermined operations. The operations can be, for example, ascrewing operation. A worker in charge of each operation cell performsan assigned operation (or assigned operations), and when theoperation(s) in the assembly operation cell is/are completed, the workerfeeds the workpiece to the next operation cell. In another embodimentthe worker may be replaced with a machine such as a robot.

Each process step of the workpiece assembly line 100 is managed by ahost system 105 in an upper layer. The host system 105 manages theprogress of each workpiece, that is, in which operation cell theworkpiece is being processed, and which operation in the operations tobe completed in the operation cell has been completed. The data relatingto the progress is transmitted to the host system 105 from eachoperation cell. When a plurality of products are manufactured, the hostsystem 105 also manages the entire progress including the number ofproducts to be manufactured, the number of completed products, and theprogress of each of the uncompleted workpieces.

A plurality of operation cells 101 through 104 are different in contentsof operation, but are the same in basic configuration. In FIG. 2, theoperation cell 103 is enlarged and shown in detail.

In the operation cell 103, a worker performs a determined operationusing a tool provided with a completion signal generation tool 110 on aworkpiece placed in a workpiece operation area 111. The completionsignal generation tool 110 generates a signal when an operation iscompleted, and is a practical example of the completion signal outputunit 1 shown in FIG. 1. An operation table can be provided for theworkpiece operation area 111.

A completion signal can be any signal capable of indicating the timingwith which an operation is completed. For example, the completion signalcan be any of an electric signal, an ultrasonic signal, and an opticalsignal, and can be transmitted by cable or by wireless. The completionsignal can be a simple pulse, or a signal indicating data of the dateand time or meaning “completion” because any signal generated when anoperation is completed can be used as a completion signal since there isno problem when the reception time of the completion signal is regardedas the time when the operation is completed. Whatever a completionsignal is, the next process, that is, the process by a positiondetection unit 112, is performed when the completion signal is received,

The position detection unit 112 includes the operation completioncoordinates detection unit 2, the operation coordinates storage unit 3,the parameter determination unit 4, and the operation position locatingunit 5 shown in FIG. 1. The position detection unit 112 receives acompletion signal from the completion signal generation tool 110, andwhen the completion signal is received, detects the operation completioncoordinates, calculates a coordinate transformation parameter, andlocates the operation position corresponding to the completed operation.A calculation unit 113 integrates the data processed in the operationcell 103, and the data transmitted and received between the operationcell 103 and the host system 105.

A display unit 114 displays various types of information for a worker.The word “display” includes both visual and audio display. That is, thedisplay unit 114 can be a display device such as a liquid crystaldisplay, a speaker, and a combination of them. The information displayedby the display unit 114 includes guidance to an operation to beperformed, the information about the operation for which the operationposition is located by the position detection unit 112, the progress ofan operation, etc.

FIG. 3 is a schematic chart showing the flow of the process in anoperation cell. In FIG. 3 as in FIG. 2, the case of the operation cell103 is described.

As shown in step S1, the host system 105 constantly performs variousprocesses during the operation of the workpiece assembly line 100. Onthe other hand, in the operation cell 103, the processes in steps S2,S3, S4, and S5 are performed in this order on each of N operations (N>1)to be performed in the process of the operation cell 103. That is, theloop of the steps S2 through S5 is repeatedly performed N times. Duringthe runtime in step S2, the information is transmitted and receivedbetween the operation cell 103 and the host system 105. In more detail,the flow of the process in FIG. 3 is described below.

When a workpiece is transmitted from the operation cell 102 for theimmediately before process to the operation cell 103 for the currentprocess, the calculation unit 113 receives data of the operationinformation from the host system 105 in step S2. The “operationinformation” refers to the information about the operation(s) to beperformed in the operation cell 103. The operation information includespredetermined information such as the number N of operations (N>1) to beperformed in the operation cell 103, the operation position and type ofeach of the N operations, etc. The operation position is represented bythe set of operation coordinates (e.g., the triplet of operationcoordinates) in the second coordinate system, and the type of operationcan be a screwing operation, a soldering operation, etc.

When step S2 is firstly performed on a certain workpiece, thecalculation unit 113 can generate the process information having thecontents that “None of N operations has been completed”, and transmit itto the host system 105. The “process information” refers to theinformation representing the progress of the operation(s) in theoperation cell 103. The process information includes the informationabout, for example, which operation in the N operations has beencompleted. In the second and subsequent operations in step S2, thecalculation unit 113 transmits to the host system 105 the processinformation about which operation has been completed immediately before.

In step S2, the calculation unit 113 further issues an instruction tothe display unit 114 to display the guidance to the operation to beperformed by the worker using voice, characters, pictures, etc. on thebasis of the updated operation information and process information.Thus, the calculation unit 113 performs various types of controlrelating to the operation in the operation cell 103. Step S3 is adisplay step performed by the display unit 114 at a given instruction.

Next, in step S4, the worker performs an operation at an instruction ofthe guidance displayed by the display unit 114. Since the present modefor embodying the invention is to process the workpiece assembly line100, the operation by the worker is an assembly operation, and ispractically, for example, a screwing operation. When one operation iscompleted, the completion signal generation tool 110 of the equipmentused for the operation generates a completion signal. The completionsignal indicates the information that one operation has been completed.

The position detection unit 112 monitors the generation of a completionsignal. In step S5, the position detection unit 112 receives acompletion signal, and upon receipt of the signal, detects the operationcompletion coordinates, determines a parameter for coordinatetransformation, performs the coordinate transformation, and locates anoperation position. As described above, the operation informationreceived by the calculation unit 113 from the host system 105 includesthe set of operation coordinates of each of the N operations. Therefore,the position detection unit 112 receives the data of the operationcoordinates from the calculation unit 113, and refers to the data,thereby locating the operation position.

Then, the position detection unit 112 transmits the information aboutthe operation whose operation position has been located to thecalculation unit 113. In the following descriptions, the informationabout the operation actually performed is also called the “operationinformation”. In step S5, the operation information transmitted from theposition detection unit 112 to the calculation unit 113 can be, forexample, a serial number (S/N) as an identifier for identification ofeach operation, a receiving time of a completion signal as thecompletion time of operation, etc.

Control is returned to step S2 after step S5. To locate the operationposition is to identify which operation has been performed in the Noperations. Thus, completing one operation corresponds to a change inprogress. Therefore, upon receipt of the operation information from theposition detection unit 112, the calculation unit 113 updates theprocess information according to the received operation information instep S2, and transmits the updated process information to both thedisplay unit 114 and the host system 105.

If the updated process information indicates that all of the Noperations to be performed in the operation cell 103 have beencompleted, then the calculation unit 113 instructs the display unit 114to display the information about the completion. The worker recognizesthe display, and transmits a workpiece to the operation cell 104 in thenext process. Otherwise, since there still remain one or more operationsto be performed in the operation cell 103, the above-mentioned processesare repeated.

As described above, the updated process information is transmitted tothe host system 105 each time the worker completes one operation. Asshown in step S1 in FIG. 2, the host system 105 manages the entireworkpiece assembly line 100, manages the process information and theoperation information about each workpiece, performs various analyses ofoperation, manages the quality of a product assembled by the workpieceassembly line 100, etc.

Next, an example of a more practical configuration of the operation cell103 is described below with reference to FIGS. 4 and 5. FIG. 4 is aperspective view of the configuration of the operation cell 103according to a mode for embodying the present invention.

The operation cell 103 is attended by a worker 201 who performs one ormore necessary operations on a workpiece 204 fixed to an operation table203. The workpiece 204 in the example shown in FIG. 4 is a printedcircuit board (Pt board) unit. For simple explanation, the followingdefinitions are provided.

N indicates the number of operations to be processed in the process ofthe operation cell 103.

All of N operations are screwing operations by a driver 202.

The driver 202 is a power-driven torque driver for which a specifiedtorque can be set. The driver 202 is configured to idle and output acompletion signal when the actually applied torque reaches a setspecified torque.

The same value of the specified torque is assigned to the screwingoperations for the N screws, and the value is set in advance in thedriver 202.

FIG. 4 shows two pictures of the driver 202. It does not show twodrivers 202, but shows the case in which the driver 202 is laidhorizontally and tightens a screw from the side, and the case in whichthe driver 202 is set vertically and tightens a screw from above. Thearrow from above the workpiece 204 toward the top surface of theworkpiece 204 and the arrow from the right side of the workpiece 204toward the right side surface of the workpiece 204 indicate the screwingdirections. Although not shown in the attached drawings, it is obviousthat there is the case in which the driver 202 is tilted and tightens ascrew.

The operation table 203 has a table plate and leg portions, and the legportions are provided with wheels so that the operation table 203 can bemoved. The table plate of the operation table 203 can be rotated on thehorizontal axis to exchange between the top surface and the reverse, andcan be rotated on the vertical axis to change the directions. Themovable operation table 203 is widely used in order to facilitate theworker 201 performing a plurality of operations without large changes ofhis/her position even if the workpiece 204 is large.

The workpiece 204 is fixed to the table plate of the operation table203. Depending on the model of product to be manufactured, the positionof the workpiece 204 fixed on the table plate of the operation table 203is predetermined. Therefore, the position and the direction of theworkpiece 204 are uniquely determined from the position and thedirection of the table plate of the operation table 203.

An antivibration camera table 205 is placed near the operation table203, and two CCD (charge coupled device) cameras 206 and 207 are placedon the camera table 205. As it is well known as the principle of stereovision, the position of a captured object can be calculated using twoimages obtained by capturing the same object from different positions. Apossible detection area 208 is an area in the operation cell 103 and isdefined as an area in which the position of an object can be calculatedby the stereo vision using the CCD cameras 206 and 207.

The range of the possible detection area 208 depends on thespecifications and the setting positions of the CCD cameras 206 and 207.Therefore, it is desired to appropriately determine the specificationsand the setting positions of the CCD cameras 206 and 207 depending onthe movable range of the operation table 203 such that the possibledetection area 208 can constantly include the workpiece 204. In thismode for embodying the present invention, the CCD cameras 206 and 207are provided diagonally above the workpiece 204 in order to enhance thedetection accuracy of the position by the stereo vision. The “WD” shownin FIG. 4 is short for “working distance”, and is used to refer to thedistance of the range in which the distance can be calculated by thestereo vision.

The operation cell 103 further includes a PC (personal computer) 209, adriver controller 210, a bar code reader 211, and an operation monitor212. The setting locations of the PC 209 and the driver controller 210are optionally determined, but they are set in the space below thecamera table 205 in FIG. 4. The operation monitor 212 is set on thecamera table 205 so that the display contents such as the guidance tothe operations to be performed can be easily checked by the worker 201.

The PC 209 performs various calculations. The PC 209 is a commoncomputer provided with, for example, a CPU (central processing unit),ROM (read only memory), RAM (random access memory), a communicationinterface for cable transmission and/or wireless communication, anexternal storage device, and a drive device of a portable storagemedium. These components are connected via a bus. The external storagedevice can be, for example, a magnetic disk device such as a hard diskdrive etc., a rewritable non-volatile memory, etc.

The driver controller 210 is connected to the driver 202 by wiring orwireless communication path not shown in FIG. 4, and receives a signaltransmitted from the driver 202. The driver controller 210 has thefunction of a driver counter of counting the number of signals receivedfrom the driver 202.

It is preferable that the bar code reader 211 is a handheld unit. Theworker 201 allows the bar code reader 211 to read the ID of the worker201, the S/N, chart numbers, etc. of the workpiece 204. The bar codeindicating the ID of the worker 201 is printed on, for example, thestaff card carried by the worker 201. The barcode indicating the S/N andthe chart number of the workpiece 204 is printed on, for example, asheet of paper to be transmitted with the workpiece 204 in each process.The paper with the bar code printed on can also be attached to theworkpiece 204.

The bar code reader 211 is connected to the PC 209 through the wiring orwireless communication path not shown in FIG. 4. The data indicated bythe bar code read by the bar code reader 211, that is, the data of theID of the worker 201 and the S/N and the chart number of the workpiece204, is transmitted to the PC 209, and transmitted from the PC 209 tothe host system 105 not shown in FIG. 4.

Since the operation table 203 is movable, it is not necessary for theworker 201 to frequently change his or her position in the operationcell 103. Therefore, in the following descriptions, it is assumed thatthere is no worker 201 between the camera table 205 and the operationtable 203. That is, it is assumed that the operation table 203 and theworkpiece 204 are taken, without being interrupted by an obstacle, inboth of the two images captured by the CCD cameras 206 and 207.

The correspondence between FIGS. 1 and 4 is described below. Thecompletion signal output unit 1 corresponds to the driver 202 and thedriver controller 210. The operation completion coordinates detectionunit 2 corresponds to the CCD cameras 206 and 207 and the PC 209. Theoperation coordinates storage unit 3 corresponds to the RAM or theexternal storage device of the PC 209. The parameter determination unit4 corresponds to the CCD cameras 206 and 207 and the PC 209. Theoperation position locating unit 5 corresponds to the PC 209.

The correspondence between FIGS. 2 and 4 is described below. Thecompletion signal generation tool 110 corresponds to the driver 202 andthe driver controller 210. The position detection unit 112 correspondsto the CCD cameras 206 and 207 and the PC 209. The calculation unit 113corresponds to the PC 209, and the display unit 114 corresponds to theoperation monitor 212.

That is, the two functions blocks, that is, the position detection unit112 and the calculation unit 113, share the PC 209 as hardware. The PC209 functions as a part of the position detection unit 112 and thecalculation unit 113 by the CPU executing the program stored in theexternal storage device or the ROM. The program can be provided througha network such as the LAN (local area network) to which the PC 209 isconnected through the communication interface or the Internet.Otherwise, the portable storage medium storing the program can be set inthe drive device, and the program can be loaded into the RAM to beexecuted by the CPU. The portable storage medium can be various types ofstorage media including an optical disk such as a CD (compact disc), aDVD digital versatile disk), etc., a magneto optical disk, a magneticdisk such as a flexible disk, etc.

FIG. 5 shows the connection of electronic equipment relating to thelocation of an operation position in the electronic equipment in theoperation cell 103 shown in FIG. 4. The connection shown in FIG. 5 canbe performed by cable and by wireless.

The CCD cameras 206 and 207, the driver controller 210, and variousexternal devices 213 including a liquid crystal display for theoperation monitor 212 are connected to the PC 209. An external device213 can also include a keyboard and a mouse for input by the worker 201.FIG. 4 shows the case in which only one driver 202 is used. As shown inFIG. 5, two drivers 202 a and 202 b can be connected to the drivercontroller 210. In some modes for embodying the present invention, it isnecessary to use a plurality of drivers depending on the type of screws.

To guarantee the necessary brightness in shooting an image using the CCDcameras 206 and 207, an illumination unit 216 not shown in FIG. 4 isused.

In the example shown in FIG. 5, the external device 213, the PC 209, thedriver controller 210, and the illumination unit 216 are connected tothe power supply of AC (alternating current) 100V. The power is suppliedto the drivers 202 a and 202 b through the driver controller 210, and tothe CCD cameras 206 and 207 through the PC 209. The path for supplyingthe power is not limited to the configuration shown in FIG. 5.

As shown in FIG. 5, the PC 209 according to the present mode forembodying the present invention is provided with an image processingboard 214 and an IO (input/output) board 215. The data of the imagecaptured by the CCD cameras 206 and 207 is received and processed by theimage processing board 214. The CPU of the PC 209 performs the processusing the result of the above-mentioned process by the image processingboard 214. The signals transmitted from the drivers 202 a and 202 b aretransmitted to the PC 209 through the driver controller 210, andreceived by the IO board 215.

The signals transmitted by the drivers 202 a and 202 b to the drivercontroller 210 are not limited to completion signals. For example, whena screwing operation fails, the worker 201 reverses the driver 202 a or202 b to loosen a screw. At this time, the driver 202 a or 202 btransmits a reverse notification signal to the driver controller 210.Then, the reverse notification information is transmitted from thedriver controller 210 to the PC 209, and received by the IO board 215.Upon receipt of the reverse notification signal, the PC 209 performs anecessary process. For example, the PC 209 locates the operationposition of the screwing operation corresponding to the loosened andremoved screw in the similar procedure performed when the completionsignal is received.

Next, with reference to FIGS. 6 through 10, the process of locating theoperation position of the completed operation is described below indetail.

FIG. 6 is a perspective view as a schematic chart showing in detail anenlarged view of a part of FIG. 4. However, in FIG. 6, the viewpoint islocated in the opposite position as compared with FIG. 4. In FIG. 4, theCCD cameras 206 and 207 are shown at the back, but they are located atthe front although not shown in FIG. 6, and the two arrows indicating anoptical axis 221 of the CCD camera 206 and an optical axis 222 of theCCD camera 207 are directed from the front to the back.

As in FIG. 4, FIG. 6 shows both the case in which the driver 202 is laidhorizontal and the case in which it is set vertically. The handleportion of the driver 202 is provided with markers 217 a and 217 b. Theattached positions of the markers 217 a and 217 b are known by attachingthem at predetermined positions or measuring the attachment positionsafter attaching them at appropriate positions. In the example below,when each marker is not to be discriminated, one or more of the markersare referred to by the reference numeral “217”.

The table plate of the operation table 203 is a rectangularparallelepiped. In the following descriptions, the surfaces of therectangular parallelepiped are referred as surfaces A, B, C, D, E, andF. In FIG. 6, the surface A is a front surface, the surface B is a rightside surface, the surface C is a back surface, the surface D is a leftside surface, the surface E is a top surface, and the surface F is abottom surface. Only the surfaces A, B, and E can be seen. Since theoperation table 203 can rotate on the vertical axis, the surface A isnot always a front surface. Since the operation table 203 can rotatealso on the horizontal axis, the surface E is not always a top surface.The workpiece 204 is fixed to the surface E or F.

Three markers 218 a, 218 b, and 218 c are attached to the surface A.Although not shown in FIG. 6, the three markers are similarly attachedto the surfaces B, C, and D. In the following descriptions, when it isnot necessary to discriminate the markers, the reference numeral “218”is used to designate one or more markers if each marker need not bediscriminated. It is predetermined in which positions the surfaces Athrough D are provided with the markers.

In FIG. 6, the workpiece 204 is fixed to a predetermined position on thesurface E. The position in which the workpiece 204 is fixed ispredetermined depending on the model of the workpiece 204. Therefore,the position and direction of the workpiece 204 can be uniquelycalculated by the position and direction of the operation table 203.

Next, the method of calculating the position and direction of the driver202 and those of the table plate of the operation table 203 from theposition of the marker calculated by the stereo vision algorithm isdescribed below. First, in preparation for the description, the terms(a) through (f) are defined below.

(a) Right Camera Coordinate System and Left Camera Coordinate System

The coordinate system of the image captured by the CCD cameras 206 and207 is respectively referred to as a “right camera coordinate system”and a “left camera coordinate system”. The right camera coordinatesystem and the left camera coordinate system are both two-dimensionalcoordinate systems. Accordingly, a point is represented by a pair ofcoordinates in these coordinate systems. Since the CCD cameras 206 and207 are mounted indifferent positions, the coordinates of an object onthe right camera coordinate system and the coordinates of the sameobject on the left camera coordinate system are different.

(b) Absolute Coordinate System

The global three-dimensional coordinate system in which the position ofthe origin and the directions of the axes are fixed is referred to as an“absolute coordinate system”, and referred to by reference characters“Σa”. The coordinates calculated by the stereo vision algorithm usingtwo images captured by the CCD cameras 206 and 207 are the coordinates(more specifically, a triplet of coordinates) on the absolute coordinatesystem Σa. The CCD cameras 206 and 207 are fixed and mounted on theantivibration camera table 205 fixed to be appropriate in calculatingthe coordinates on the absolute coordinate system Σa.

In the present mode for embodying the present invention, the absolutecoordinate system Σa is a three-dimensional orthogonal coordinatesystem, and the three axes are an x axis, a y axis, and a z axis. Theabsolute coordinate system Σa is a practical example of the firstcoordinate system described with reference to FIG. 1. The triplet ofcoordinates on the absolute coordinate system Σa representing theposition of an optional point P_(i) is expressed by the row vector^(a)p_(i) of the equation (1).

^(a) p _(i)=(x _(i) ,y _(i) ,z _(i))  (1)

(c) Operation Table Coordinate System

The local three-dimensional coordinate system on the basis of the tableplate of the operation table 203 is called an “operation tablecoordinate system”. The operation table coordinate system is acoordinate system in which the origin moves with the movement of theoperation table 203, and the directions of the axes change with therotation of the table plate of the operation table 203.

(d) Workpiece Coordinate System

The local three-dimensional coordinate system on the basis of theworkpiece 204 is called a “workpiece coordinate system”, and referred toby reference characters “Σw”. The workpiece coordinate system is acoordinate system in which the origin moves with the movement of theworkpiece 204, and the directions of the axes change with the rotationof the workpiece 204.

In the present mode for embodying the present invention, the workpiececoordinate system Σw is a three-dimensional orthogonal coordinatesystem, and the three axes are a q axis, an r axis, and an s axis. Theworkpiece coordinate system Σw is a practical example of the secondcoordinate system described with reference to FIG. 1. The triplet ofcoordinates on the workpiece coordinate system Σw of the point P_(i) isexpressed by the row vector ^(w)p_(i) of the equation (2).

^(w) p _(i)=(q _(i) ,r _(i) ,s _(i))  (2)

(e) Coordinate Transformation

In each of the above-mentioned coordinate systems, (a) and (c) areauxiliary coordinate systems, and (b) through (d) can be mutuallytransformed, but the present mode for embodying the present invention isfocused on the transformation from the absolute coordinate system Σa tothe workpiece coordinate system Σw. The transformation from the absolutecoordinate system Σa to the workpiece coordinate system Σw can beexpressed by the equation (3) using the 4×4 matrix A. “T” is a symbolrepresenting the transpose.

(q _(i) ,r _(i) ,s _(i),1)^(T) =A(x _(i) ,y _(i) ,z _(i),1)^(T)  (3)

Each element of the matrix A is a parameter for the coordinatetransformation from the absolute coordinate system Σa to the workpiececoordinate system Σw, and is a practical example of a necessaryparameter for the coordinate transformation between the first coordinatesystem and the second coordinate system described with reference toFIG. 1. As described above, since the operation table 203 to which theworkpiece 204 is fixed is movable, each value of the element of thematrix A dynamically changes depending on the position and direction ofthe operation table 203.

(f) Distance Between Two Points

The Euclid distance between the points P_(i) and P_(j) is expressed by“d(P_(i), P_(j))”. The coordinate systems (b) through (d) are coordinatesystems for a three-dimensional space in the real world, and thedistance is the same when calculated in any coordinate system.

Next, using the terms defined as described above, the process performedin the operation position locating system shown in FIGS. 2 and 4 isdescribed in detail with reference to FIGS. 7 through 10. FIG. 7 is aflowchart showing the flow of the process according to a mode forembodying the present invention. As similar as FIG. 2, the followingdescription is focused on the process in the operation cell 103.

First, in step S101, the workpiece 204 is transmitted to the operationcell 103 from the operation cell 102 in the process immediately before.By the worker 201 allowing the bar code reader 211 to read the bar coderepresenting the model of the workpiece 204, the PC 209 functioning asthe calculation unit 113 detects that the workpiece 204 has beentransmitted, and is informed of the model of the workpiece 204. The PC209 transmits a code indicating the model as a retrieval key (namely,search key) to the host system 105, and reads the operation informationabout the model from the host system 105. The operation informationincludes the coordinates on the workpiece coordinate system Σw at thescrewing position of N screws. These coordinates are determined when theworkpiece 204 is designed, and are the operation coordinates describedabove with reference to FIG. 1.

Next, in step S102, the PC 209 monitors whether or not a completionsignal has been received. The process in step S102 is repeatedlyperformed until a completion signal is received. When the completionsignal is received, control is passed to step S103. A practical exampleof the completion signal is a torque-up signal indicating that theactually applied torque has reached a specified torque. The torque-upsignal is transmitted from the driver 202 to the PC 209 through thedriver controller 210 and the IO board 215.

In step S103, the CCD cameras 206 and 207 simultaneously capture theoperation cell 103. The data of the two captured images is fetched tothe image processing board 214 in the PC 209. Thus, the necessary datain detecting the positions and the directions of the workpiece 204 andthe driver 202 at the moment when the reception of the completion signalis detected is reserved.

Next, in step S104, on each of the two images captured by the CCDcameras 206 and 207, the image processing board 214 detects thepositions of the markers 218 attached to the operation table 203 in theimage recognizing process.

As described above, the images captured by the CCD cameras 206 and 207includes the operation table 203 and the workpiece 204 without anyobstacle such as the worker 201 etc. Therefore, at least one of thesurfaces A through D is captured in the image with the three attachedmarkers 218. Thus, the image processing board 214 is expected to detectat least three markers 218 in the data of each of the two images.

In the following description, for explanation convenience, thepositional relation shown in FIG. 6 is assumed, and it is assumed thatthe markers 218 a through 218 c have been detected in step S104.However, at the stage in steps S104 and S105, it is not determined thatthe detected three markers 218 are the markers 218 a through 218 cattached to the surface A.

In the descriptions below, the respective central positions of themarkers 218 indicate the respective positions of the markers 218. Instep S104, the coordinates in the right camera coordinate system andthose in the left camera coordinate system at the central positions ofthe markers 218 a through 218 c are detected.

In some cases, two adjacent surfaces such as the surfaces A and B appearin an image, and the image processing board 214 can detect six markers218. On each surface, since the markers 218 are attached on one straightline, three markers 218 are arranged on a straight line in each image,and other three markers 218 are arranged on another straight line. Theimage processing board 214 detects these two straight lines so that thesix markers 218 can be grouped into those attached to one surface andthose attached to another surface. Only one group can be used or bothgroups can be used in the following processes in steps S105 and S106. Inthe following descriptions, it is assumed for simple explanation thatonly one group including the markers 218 a through 218 c is used.

In the next step S105, the PC 209 calculates the coordinates in theabsolute coordinate system Σa of each of the detected markers 218 by thestereo vision algorithm. The calculated values of the coordinates in theright camera coordinate system and the left camera coordinate system ofeach of the markers 218 obtained in step S104 are used in thecalculation in step S105. That is, in this example, the coordinates^(a)p₁, ^(a)p₂, ^(a)p₃ of the absolute coordinate system Σa of each ofthe respective positions P₁, P₂, P₃ of the markers 218 a, 218 b, and 218c are calculated.

In the next step S106, the PC 209 calculates the position and thedirection of the operation table 203 represented by the absolutecoordinate system Σa. The calculating method is described below withreference to FIGS. 8 and 9.

FIG. 8 is a schematic chart of the positional relation among thedetected three markers 218 a through 218 c. In step S106, the PC 209calculates the distance among the markers 218, that is, d(P₁, P₂) andd(P₂, P₃), using the coordinates calculated in step S105. FIG. 8 showsthe two distances of d(P₁, P₂) and d(P₂, P₃).

Three markers 218 are attached to each of the surfaces A through D.These markers 218 are attached such that the positional relation amongthe markers 218 can be different for each surface, and also differentdepending on which is the upper surface, the surface E or the surface F.FIG. 9 shows an example of an arrangement pattern of the markers 218.

In FIG. 9, it is assumed that the surfaces E and F are squares eachhaving a side of 105 cm, and each of the markers 218 has a horizontallength of 5 cm. In each of the surfaces A through D, the markers 218 areattached to both right and left ends, and another marker 218 is attachedbetween them. Therefore, on each of the surfaces A through D, thedistance between the two markers 218 attached to both ends is 100 cm.FIG. 9 shows the positions of the three markers 218 by the referencecharacters P₁, P₂, and P₃ in this order from the left of the image. Thetable shown in FIG. 9 shows, in order from the left, four columns of thenominal value of the predetermined d(P₁, P₂), the nominal value of thepredetermined d(P₂, P₃), the surface to which the markers 218 detectedin step S104, and the top surface.

Based on the above-mentioned assumption, the PC 209 determines, forexample, if the value of the calculated d(P₁, P₂) can be taken equal to10 cm within a predetermined permissible error, and if the value of thecalculated d(P₂, P₃) can be taken equal to 90 cm within a predeterminedpermissible error, then the surface to which the three markers 218detected in step S104 are attached is the surface A, and the surface Eis a top surface.

Otherwise, it is not necessary to predetermine a permissible error. Inthis case, the PC 209 can determine, for example, if the value closestto the calculated value d(P₁, P₂) in the leftmost column of the tableshown in FIG. 9 is 10 cm, and the value closest to the calculated valued(P₂, P₃) in the second column from the leftmost column of the tableshown in FIG. 9 is 90 cm, then the surface to which the three markers218 detected in step S104 are attached is the surface A, and the surfaceE is a top surface.

Thus, in comparing the values of the calculated distances among themarkers 218 with the values of the predetermined distances among themarkers 218, a process is performed with the error taken into account,but a practical processing method is optionally determined depending onmodes of embodiments.

The table shown in FIG. 9 includes 8 rows excluding the header linebecause each of the four surfaces A through D has the case in which thesurface E is a top surface, and the case in which the surface F is a topsurface. The markers 218 are arranged such that in any two of the eightrows the combination of the values of the d(P₁, P₂) and the d(P₂, P₃)cannot be the same.

The predetermined arrangement pattern of the markers 218 such as shownin FIG. 9 is read to the RAM etc. of the PC 209 in advance. Therefore,the PC 209 determines the surface to which the detected three markers218 are attached and the top surface based on the calculated values ofd(P₁, P₂) and d(P₂, P₃) with reference to the arrangement pattern storedin advance. For example, in the example shown in FIG. 6, the PC 209recognizes that the surface to which the three image-recognized markers218 are attached is the surface A, the top surface is the surface E, andthe three markers 218 are practically the markers 218 a through 218 cattached to the surface A.

The recognizing process is performed after the PC 209 has calculated thecoordinates ^(a)p₁, ^(a)p₂, ^(a)p₃ in the absolute coordinate system Σaof the positions P₁, P₂, P₃ of the markers 218 a through 218 c.Therefore, the PC 209 can calculate the position and direction of theoperation table 203 represented by the absolute coordinate system Σabased on the coordinates ^(a)p₁, ^(a)p₂, and ^(a)p₃. For example, the PC209 calculates the position of the origin and the direction vectors ofthe axes of the operation table coordinate system on the absolutecoordinate system Σa.

In the next step S107, the PC 209 calculates the matrix A as a transformparameter for transformation between the coordinate systems by using theposition and direction of the operation table 203.

First, the PC 209 calculates the matrix B as a parameter for coordinatetransformation from the absolute coordinate system Σa to the operationtable coordinate system based on the calculation result in step S106.

On the other hand, since the workpiece 204 is fixed to a predeterminedposition on a predetermined surface of the table plate of the operationtable 203, the matrix C as a parameter for coordinate transformationfrom the operation table coordinate system to the workpiece coordinatesystem Σw is statically determined in advance depending on the model ofthe workpiece 204, and does not change. The matrix C can be stored inadvance on the hard disk drive of the PC 209, for example, or can beprovided to the PC 209 from the host system 105 in step S107.

Therefore, by performing a matrix operation to calculate A=CB, the PC209 can obtain the matrix A of the equation (3).

Alternatively, the PC 209 can also calculate the matrix A as follows.The coordinates in the operation table coordinate system of each of themarkers 218 are predetermined. The coordinates of the markers 218 areread in advance to, for example, the RAM of the PC 209. The PC 209 cantransform the read coordinates of the markers 218 into the coordinatesin workpiece coordinate system Σw using the matrix C.

At this stage, the coordinates both in the absolute coordinate system Σaand in the workpiece coordinate system Σw of the positions P₁ through P₃of the markers 218 a through 218 c have been calculated. Therefore, thePC 209 can calculate the matrix A based on the coordinates in coordinatesystems Σa and Σw.

After the execution in step S107, control is passed to step S108. Instep S108, the image processing board 214 detects the markers 217 a and217 b of the driver 202 by image recognition processing for each of twoimages captured in step S103. That is, the coordinates in the rightcamera coordinate system and those in the left camera coordinate systemof the markers 217 a and 217 b are detected.

In the next step S109, the PC 209 calculates the coordinates in theabsolute coordinate system Σa of the markers 217 a and 217 b by thestereo vision algorithm. This is a similar process performed in stepS105.

In the next step S110, the PC 209 calculates the coordinates in theabsolute coordinate system Σa of a tip 225 of the driver 202 using thecoordinates of the markers 217 a and 217 b calculated in step S109. Thecalculating method is described below with reference to FIG. 10. FIG. 10is a schematic chart represented in two-dimensional representation.

The two markers 217 a and 217 b are attached to the handle portion ofthe driver 202. For simple explanation, it is assumed that the twomarkers 217 a and 217 b are attached to the position that cannot behidden behind the hands of the worker 201. The driver 202 has a handleportion and a driver bit 223 that form a substantially radial symmetryabout an axis 224. The markers 217 a and 217 b are also respectivelyattached to form a radial symmetry about the axis 224. FIG. 10 shows theoperation of tightening a screw 226 at a predetermined position of theworkpiece 204, and the position of the tip 225 of the driver 202 whenthe screw 226 is completely tightened is the operation completionposition of the operation.

In the process in step S109, the coordinates ^(a)p₄ and ^(a)p₅ in theabsolute coordinate system Σa of the points P₄ and P₅ indicating thepositions of the markers 217 a and 217 b are calculated. The point P₄indicates the marker 217 a, and the method of selecting the point P₄ isoptionally determined depending on the mode for embodying the presentinvention.

For example, in the mode for embodying the present invention where thediameter of the handle portion of the driver 202 is sufficiently lowerthan the distance between the operation positions, there is no problemoccurring when the thickness of the driver 202 is ignored. Otherwise,there is a mode for embodying the present invention in which thediameter of the driver 202 is lower than a predetermined permissibleerror depending on the accuracy of the coordinates in the absolutecoordinate system Σa calculated based on the image captured by the CCDcameras 206 and 207. In the above-mentioned modes for embodying thepresent invention, when a straight line passing through a point at thecenter of the width of the marker 217 a and corresponding to the axis224 in the captured image is assumed, a point located on the straightline can be selected as a point P₄. A point P₅ indicating the positionof the marker 217 b is similarly selected. Thus, the coordinates of thetip 225 is calculated under an assumption that the tip 225 of the driver202 falls on the straight line connecting the point P₄ to the point P₅.

In another mode for embodying the present invention, for example, anintersection point of the plane perpendicular to the axis 224 passingthrough the center of the width of the marker 217 a and the axis 224 canbe selected as a point P₄. In this case, since the point P₄ is not onthe outer surface of the marker 217 a, the PC 209 can calculate thecoordinates of the point P₄ from the coordinates of a plurality ofpoints on the outer surface of the marker 217 a calculated based on thestereo vision principle, and the known diameter of the handle portion ofthe driver 202.

In the following descriptions, it is assumed that the straight lineconnecting the point P₄ indicating the marker 217 a and the point P₅indicating the marker 217 b can be regarded as the axis 224. Based onthe assumption, the point P₆ of the tip 225 of the driver 202 is locatedon the axis 224.

In the absolute coordinate system Σa, the equation indicating thestraight line of the axis 224 can be obtained based on the coordinates^(a)p₄ and ^(a)p₅. The angle θ shown in FIG. 10 is an angle made by theworkpiece 204 and the axis 224 and represented in a two-dimensionalschematic chart.

On the other hand, the attachment position of the markers 217 a and 217b on the driver 202 is known as described above. The length of thedriver bit 223 is predetermined or measured in advance. Therefore, it ispossible to calculate the distance d(P₅, P₆) between the point P₅ andthe point P₆ from the known static data only.

The PC 209 calculates the coordinates of the position of the distanced(P₅, P₆) from the point P₅ in the opposite direction of the point P₄ asviewed from the point P₅ on the straight line of the axis 224 in stepS110. The PC 209 then determines the calculated coordinates as thecoordinates of the point P₆ in step S110.

Which is the marker 217 a (or 217 b) between the two image-recognizedmarkers can be determined in various methods, and the coordinates of thepoint P₆ are calculated based on the determination.

For example, in a mode for embodying the present invention, theoperation of tightening the screw 226 with the driver 202 setperpendicularly is limited to the case in which the screw 226 is placedand the driver 202 tightens the screw from above. Therefore, when thetilt of the straight line of the axis 224 is nearly perpendicular, ahigher marker of the two position-detected markers is recognized as themarker 217 a, and a lower marker is recognized as the marker 217 b.

There is also a mode for embodying the present invention in which theposition of the worker 201 is limited to a predetermined range, forexample, it is predetermined that the worker 201 is positionedconstantly to the left of the workpiece 204 as viewed from the CCDcameras 206 and 207. In this case, in the two position-detected markers,the left marker as viewed from the CCD cameras 206 and 207 can berecognized as the marker 217 a, and the right marker can be recognizedas the marker 217 b.

Otherwise, the feature in geometry that the driver 202 has a thickhandle portion and the thin driver bit 223 can be taken into account. Itis also possible to recognize a marker farther from the driver bit 223as the marker 217 a in the two coordinates-detected markers, and acloser marker as the marker 217 b.

In any case, the PC 209 calculates the coordinates ^(a)p₆ in theabsolute coordinate system Σa of the point P₆ indicating the tip 225 ofthe driver 202 based on the coordinates of the points P₄ and P₅indicating the markers 217 a and 217 b. The coordinates ^(a)p₅ arepractical example of the operation completion coordinates described withreference to FIG. 1.

Then, the PC 209 coordinate-transforms in step S111 the coordinates^(a)p₆ using the parameter calculated in step S107, that is, the matrixA, and calculates the coordinates ^(w)p₆ in the workpiece coordinatesystem Σw of the point P₆.

Afterwards, in step S112, the PC 209 compares the design data of thescrewing position read in step S101 as operation coordinates with thecoordinates ^(w)p₆ of the tip 225 of the driver 202 calculated in stepS111, and locates (namely, identifies) to which screwing position thejust completed operation corresponds. The coordinate system used in theprocess in step S112 is the workpiece coordinate system Σw only.

For example, from the coordinates ^(w)p₆ and N operation coordinates,the PC 209 calculates the distance between the point P₆ and each of theN operation positions, thereby N values of distances being obtained.Then, the PC 209 locates the operation position indicating the shortestdistance from the point P₆ as the operation position of the justcompleted screwing operation. Alternatively, the coordinates ^(w)p₆ arenot compared with all of the N triplets of operation coordinates, but atriplet of the operation coordinates of the completed operation can bedeleted from the comparison targets.

In the next step S113, the PC 209 determines whether or not the locationin step S112 has succeeded. If it has succeeded, control is passed tostep S114, and the PC 209 controls the operation monitor 212 to displaythe location result. If it has failed, control is passed to step S115,and the PC 209 controls the operation monitor 212 to display that thelocation has failed.

The criterion in step S113 depends on the mode for embodying the presentinvention. For example, assume a case in which any of the N operationpositions that indicates the shortest distance to the tip 225 of thedriver 202 in the N operation positions is selected in step S112. Inthis case, only if the shortest distance is equal to or lower than apredetermined threshold, then it is determined in step S113 that thelocation has succeeded. Otherwise, it can be determined that thelocation has failed.

Although not shown in FIG. 7, there can be the case in which imagerecognition fails in step S104 or S108. In this case, since theoperation position cannot be located accordingly, it is determined “No”in step S113.

After performing step S114 or S115, control is passed to step S116. Instep S116, the PC 209 determines whether or not all of the N operationsto be performed in the operation cell 103 have been completed. If all ofthem have been completed, it determines “Yes”, and the process shown inFIG. 7 terminates. If there still remains any operation to be performed,the PC 209 determines “No”, and control is returned to step S102. Whenthe process shown in FIG. 7 is completed, the workpiece 204 istransmitted to the next operation cell 104.

FIG. 11 shows an example of the display screen of the operation monitor212. In the example shown in FIG. 11, nine areas are provided on thescreen of the operation monitor 212, and the nine types of informationare displayed by a character, symbol, picture, etc. in the correspondingareas.

A workpiece selecting operation display area 301 is an area forselection and display of a model of the workpiece 204 being assembled.In a mode for embodying the present invention, the worker 201 selects amodel of the workpiece 204 from among the options displayed on theworkpiece selecting operation display area 301 and indicates theselection result on the PC 209 through the external device 213, and theselection result is displayed on the workpiece selecting operationdisplay area 301. In another mode for embodying the present invention,the result of the worker 201 allowing the bar code reader 211 to readthe bar code indicating the model of the workpiece 204 is displayed onthe workpiece selecting operation display area 301. Therefore, theworker 201 only has to confirm the displayed model.

An operation information display area 302 displays the S/N of theworkpiece 204 and the ID of the worker 201. A message for prompting theworker 201 to input his or her ID using the bar code reader 211 can alsobe displayed.

An operation button display area 303 includes “start”, “terminate”,“suspend”, and “resume” buttons displayed. The worker 201 depresses the“start” button when a series of operations for the workpiece 204 isstarted using, for example, a mouse, depresses the “terminate” buttonwhen a series of operations for the workpiece 204 is forcibly terminatedfor any reason, depresses the “suspend” button when a series ofoperations is to be suspended for a break etc., and depresses the“resume” button when a series of suspended operations is resumed.

A detail display area 304 includes detailed information such as thestatus of the entire system, the progress of N operations to beperformed in the operation cell 103, various messages including theinstructions for the operation to be performed next, etc.

The workpiece 204 selected in the workpiece selecting operation displayarea 301 is graphically displayed in a workpiece face display area 305and a workpiece reverse display area 306. In the example shown in FIG.11, it is assumed that an operation for both surfaces of face andreverse sides of the workpiece 204 is required. Therefore, the workpieceface display area 305 displays the face of the workpiece 204, and theworkpiece reverse display area 306 shows the reverse of the workpiece204. The operation-completed portions and the portions to beoperation-performed are graphically displayed. For example, theoperation-completed portions and the portions to be operation-performedare respectively marked with black and white circles. Theoperation-completed portions refer to the portions in which the locationof an operation position has been successfully performed.

A screwing count display area 307 includes the number of times Mindicating the frequency of the determination “Yes” in step S113 shownin FIG. 7, namely, the number of times M indicating the frequency of thesuccessful locating of the operation position. Or, the value (N−M)obtained by subtracting the number of times M from the total number N ofthe operations to be performed can be displayed in the screwing countdisplay area 307.

A total screwing number display area 308 includes the total number N ofthe operations to be performed in the operation cell 103 on theworkpiece 204.

A locating error count display area 309 includes the number of “No” instep S113 shown in FIG. 7, that is, the number of times of unsuccessfuloperations of locating an operation position for any reason.

FIG. 12 shows an example of the marker 218 attached to the operationtable 203. The marker 218 is provided with a sheet form base unit 401and a cross-shaped projection 402.

In the example shown in FIG. 12, the geometry of the sheet form baseunit 401 is the form of a rectangular sheet. A reverse 404 of the sheetform base unit 401 is an attachment surface to the operation table 203.For example, an adhesive is applied to the reverse 404, and the marker218 is attached to the operation table 203. The cross-shaped projection402 is extended in the thickness direction of the sheet form base unit401 from the face 403 of the sheet form base unit 401. The cross-shapedprojection 402 includes a horizontal line portion 405 and a verticalline portion 406 orthogonal to each other and having the width of w1 andthe thickness of t1. The sheet form base unit 401 is made of a highreflectance material such as a retroreflective material etc. for causingretroreflection indicating the reflection of light in the direction ofincident light. The cross-shaped projection 402 is made of a lowreflection material having a low reflectance of incident light.

With the configuration of the markers 218, the accuracy of detection ofthe markers 218 by image recognition performed by the image processingboard 214 on the images captured by the CCD cameras 206 and 207 can beenhanced for the following first through third reasons.

First, since the sheet form base unit 401 is made of a retroreflectivematerial, the reflectance is substantially constant regardless of thecapture angle of the CCD cameras 206 and 207, and is high. On the otherhand, the cross-shaped projection 402 is made of a low reflectancematerial. Therefore, regardless of the capture angle, a high contrastcross-shaped image is taken in the captured image, and the outline ofthe image is clear. Thus, the recognition accuracy for recognizing themarkers 218 can be enhanced when the markers 218 are recognized by imagerecognition using a technique such as the well-known edge extractingmethod.

Second, since there are few high contrast cross-shaped objects in anoperation cell in many cases, the markers 218 are not mixed with thebackground of the image, and therefore can be easily detected. Theimages captured by the CCD cameras 206 and 207 include various objectsin the operation cell and the worker 201. Many of them have circular orrectangular outlines. Therefore, it is empirically informed that therecognition rate of markers is degraded by using markers with circularpatterns, rectangular patterns, or the like. On the other hand, therecognition rate of the marker 218 shown in FIG. 12 is hardly degradedin undesired conditions such as various objects included in thebackground.

Third, by appropriately determining the width w1 and the thickness t1 ofthe cross-shaped projection 402, it is possible to reduce the change inthe width of the horizontal line and the vertical line of the crossshape in an image regardless of the angle of capturing the marker 218.

When the marker 218 is captured from the front, what is captured in theimage is substantially only the front side of the cross-shapedprojection 402, and most of the side surfaces of the horizontal lineportion 405 and the vertical line portion 406 are not captured.Therefore, in this case, the width of the horizontal line in the imageis determined by the distance between the CCD camera 206 or 207 and themarker 218 and the width w1 of the horizontal line portion 405. The sameholds true with the width of the vertical line in the image.

On the other hand, when the marker 218 is captured diagonally from theleft or right of them, both of a front surface and a side surface of thevertical line portion 406 are captured in the image. At a capture angleat which the width of the front surface in the image is the narrower,the width of the side surface in the image is the broader. Therefore, ifthe width w1 and the thickness t1 of the cross-shaped projection 402 areappropriate, the width of the vertical line of the cross shape in theimage does not largely change depending on the capture angle. For asimilar reason, when the marker 218 is captured from diagonally above orbelow them, the width of the horizontal line of the cross shape in theimage does not largely change depending on the capture angle.

When the change of the width of the vertical line and the horizontalline of the cross shape is small in the image, the cross shapecorresponding to the marker 218 in the image can be easily recognized,and the accuracy of the image recognition can be naturally enhanced.Since the appropriate width w1 and thickness t1 depend on variouselements such as the size of an operation cell, the mounting angle ofthe CCD cameras 206 and 207, the size of the markers 218, etc., it isdesired to appropriately determine them after experiments etc.

FIGS. 13A and 13B show examples of parts to be attached to the driver202 for use as the marker 217. In FIGS. 6 and 10, two markers 217 a and217 b are shown without limiting a practical geometry or material, butthey do not refer to two different objects required. It is obvious thatone marker 217 as an object, which has two position recognizableportions, also can be used. FIGS. 13A and 13B show examples of themarker 217.

The marker 217 shown in FIG. 13A is attached to the handle portion ofthe driver 202. It is desired that the marker 217 is attached to theposition in which it is not hidden by the hands of the worker 201. Themarker 217 includes the cylindrical base unit 408 and three cylindricalprojections 409 a, 409 b, and 409 c. The cylindrical base unit 408 ismade of a high reflectance material as with the sheet form base unit401. The cylindrical projections 409 a, 409 b, and 409 c are made of alow reflectance material as with the cross-shaped projection 402.

Each of the cylindrical projections 409 a, 409 b, and 409 c is mountedalong the perimeter outside the side surface of the cylinder of thecylindrical base unit 408. That is, the cylindrical base unit 408 andthe cylindrical projections 409 a, 409 b, and 409 c are configured toform a radial symmetry on a common axis 410, and the entire marker 217forms the radial symmetry on the axis 410. The length in the directionof the axis 410 is called as a “width” in the description on FIG. 13A.In the example shown in FIG. 13A, the widths of the cylindricalprojections 409 a, 409 b, and 409 c are w2. The length in the radialdirection, that is, the distance from the axis 410 is called a“thickness” in the description on FIG. 13A. The cylindrical projections409 a, 409 b, and 409 c are extended outside by the thickness of t2 fromthe cylindrical base unit 408.

The marker 217 shown in FIG. 13A is attached to the driver 202 such thatthe axis 410 of the marker 217 and the axis 224 of the driver 202 canmatch each other. For example, the cylindrical base unit 408 is hollowand is formed such that its inner diameter is substantially equal to theouter diameter of the handle portion of the driver 202. In this case, byengaging all or a part of the handle portion of the driver 202 in thespace inside the cylindrical base unit 408, the marker 217 can beattached to the driver 202 such that the axis 410 can match the axis224. Otherwise, the cylindrical base unit 408 that is not hollow butsolid is acceptable. In this case, for example, the marker 217 can beattached to the driver 202 by attaching the cylindrical projection 409 cto the end of the handle portion of the driver 202. In addition, a partsuch as a pawl for attachment to the driver 202 can be provided for themarker 217.

FIGS. 6 and 10 show the driver 202 with two markers 217 a and 217 bbecause at least two objects whose coordinates are to be detected arerequired. To reduce detection error, it is obvious that three or moreobjects whose coordinates are to be detected can be used. Therefore,FIG. 13A shows the marker 217 provided with three cylindricalprojections 409 a, 409 b, and 409 c whose coordinates are to bedetected.

With the marker 217 configured as described above, the accuracy of theimage recognition of the marker of the driver by the PC 209 on the imagecaptured by the CCD cameras 206 and 207 can be enhanced for thefollowing reasons.

First, since the cylindrical base unit 408 is made of a retroreflectivematerial, the contrast between the cylindrical base unit 408 and thecylindrical projections 409 a, 409 b, and 409 c in the captured image isconstantly high regardless of the capture angle. Therefore, regardlessof the capture angle, the cylindrical projections 409 a, 409 b, and 409c can be easily detected by image recognition.

Second, since the marker 217 takes the form of a radial symmetry, thegeometry of the marker 217 in the captured image does not greatly changeeven if a slope of a line passing through the marker 217 and the CCDcamera 206 (or CCD camera 207) largely changes on the horizontal planeparallel to the floor. The feature contributes to the improvement inrecognition rate.

Third, since the cylindrical projections 409 a, 409 b, and 409 c havethe thickness of t2, and are extended outside the cylindrical base unit408, the cylindrical projections 409 a, 409 b, and 409 c having acertain degree of size are captured in the image even if the angle madeby the optical axis of the CCD camera 206 or 207 and the axis 410 of themarker 217 is small. The third reason is similar to the third reasonrelating to the marker 218 shown in FIG. 12. Therefore, by appropriatelydetermining the width w2 and the thickness t2 depending on the mode forembodying the present invention, the range in which the marker 217 canbe detected by the image recognition can be expanded, thereby enhancingthe recognition rate.

The marker 217 shown in FIG. 13B has almost the same configuration asthe marker 217 shown in FIG. 13A. Only the difference is that the marker217 shown in FIG. 13B is further provided with a cylindrical projection409 d.

As exemplified in FIGS. 13A and 13B, a marker can be provided to bepreferably used in the operation position locating system. The marker isattachable to a device such as the driver 202, and comprises a base unitand a plurality of projections extending from the base unit. The baseunit has high reflectance and the plurality of projections have lowreflectance, or the base unit has low reflectance and the plurality ofprojections have high reflectance. Each of the plurality of projectionstakes a form of a substantially radial symmetry about an axis commonamong the plurality of projections when the marker is attached to thedevice.

Since the projections of the marker take the form of substantiallyradial symmetry, the position of the marker can be detected withconstant reliability when the tool such as the driver 202 is observedfrom any direction by attaching the marker to the tool for use in themanufacturing process. Using the above-mentioned marker, the detectionaccuracy of the operation completion coordinates detection unit can beimproved.

Described above in detail are the modes for embodying the presentinvention. The summary of the effect obtained by above-describedembodiments relating to the operation cells 101 through 104 in theworkpiece assembly line 100 is described below.

First, the operations to be performed can be divided into those actuallyperformed and those not performed yet. Therefore, it is possible, forexample, for the PC 209 or the host system 105 to automatically checkwhether or not there is any missed operation not performed in themanufacturing process. By the worker 201 performing the operation basedon the check result, the workpiece 204 in the state in which there is anoperation to have been performed, but has not been performed yet can beprevented from being erroneously transmitted to the next process.

Therefore, it is possible to reduce the number of man-hours for anoperation instruction to be provided for the worker 201 who has failedto perform an operation. Furthermore, the number of man-hours in testingprocess steps to be performed after the manufacturing process can alsobe largely reduced. Conventionally, it has been necessary that aninspector visually sees an appearance of the workpiece 204 to search forany portion of missing an operation, and to check the location of themissed operation. However, according to the above embodiments, it is notnecessary to perform the inspection, or it is possible to largely reducethe inspection steps.

In addition, according to the above embodiments, the number ofoccurrences of defective products due to forgetting an operation can bereduced, thereby enhancing the yield of products.

In addition to the above-mentioned common effects, the modes forembodying the present invention described above in detail have thefollowing effect.

First, without using a marker requiring electric power, and using themovable operation table 203, an operation position can be located.Therefore, the operation position locating system according to theabove-mentioned modes for embodying the present invention can be appliedto a larger number of manufacturing processes.

Furthermore, since the operation position locating system is providedwith the function of providing guidance for the worker 201, and variouspieces of information shown in FIG. 11 are displayed on the operationmonitor 212, the worker 201 can perform an operation while checking theoperation monitor 212, and can be prevented from forgetting tightening ascrew. Furthermore, even if the worker 201 carelessly forgets to tightenthe screw, it is displayed on the operation monitor 212, the worker 201recognizes the failure, and then the worker 201 performs the operationcorrectly.

Since the driver 202 is an electrically driven torque driver, the torqueapplied to a screw and the driving time can be detected. Therefore,scoring (or galling) of a screw is detectable, and only placing a screwand forgetting tightening it can also be detected.

For example, when a torque is correctly applied in the direction of thescrew hole, the time taken from starting tightening a screw to reachinga specified torque is expected to be equal to the time taken fromstarting tightening the screw to correctly setting the screw. Therefore,when the difference between the time values is large, it is detectablethat scoring the screw has occurred because the torque has been appliedto the screw inclined against the screw hole. When the scoring of ascrew is detected, it is desired that the driver 202 issues a signal asa notification of a defective operation instead of issuing a completionsignal. According to the signal, the PC 209 controls the operationmonitor 212 to display a warning in the detail display area 304 etc.,thereby allowing the worker 201 to perform an appropriate operation.

If a screw is forgotten after being placed in its position, thecompletion signal corresponding to the screwing operation is not outputfrom the driver 202. Therefore, the operation position locating systemrecognizes that the operation has not been performed. Accordingly, theoperation position of the operation is displayed as a portion for whichan operation is to be performed in the workpiece face display area 305or the workpiece reverse display area 306, thereby allowing the worker201 to recognize that he/she forgot to tighten the screw.

The present invention is not limited to the above-mentioned modes forembodying the present invention, but variations can be applied.Described below are examples of the variations.

In the above-mentioned modes for embodying the present invention, anoperation completion position is compared with an operation position byrepresenting them by coordinates in workpiece coordinate system Σw inorder to locate the operation position corresponding to the operationcompletion position. On the other hand, they can be compared byrepresenting them by the coordinates in the absolute coordinate systemΣa. In this case, in step S107 shown in FIG. 7, the matrix A is replacedwith a matrix A⁻¹ in calculation. The matrix A⁻¹ is an inverse matrix ofthe matrix A, and a transformation matrix from the workpiece coordinatesystem Σw to the absolute coordinate system Σa. In this case, theprocess in step S111 is unnecessary, and a process of transforming theoperation coordinates into the coordinates in the absolute coordinatesystem Σa in step S112 is required.

Although it is possible to adopt the above-mentioned variation, it isnot so preferable. That is, the method of transforming the coordinatesin the absolute coordinate system Σa into those in the workpiececoordinate system Σw as shown in FIG. 7 requires less computationalcomplexity and is therefore efficient.

In FIG. 7, only one set of coordinates representing one point istransformed from the absolute coordinate system Σa to the workpiececoordinate system Σw in step S111. On the other hand, if the number ofoperations to be performed in the operation cell 103 is N, it isnecessary to transform N sets of operation coordinates from theworkpiece coordinate system Σw into the absolute coordinate system Σa inthe variation example described above. In this case, the larger thevalue of N is, the higher the computational complexity is. In some modesfor embodying the present invention, the already-located operationposition can be removed from a comparison target with the operationcompletion position. Nevertheless, the variation example requires highercomputational complexity than the case shown in FIG. 7.

Since the operation table 203 is movable, it is necessary to transformthe coordinates in one coordinate system to another coordinate systemeach time an operation is completed. Therefore, the method of firstperforming a coordinate transformation, storing the transformationresult, and then referring to the stored transformation result cannot beused. From this viewpoint, it is clear that the computational complexityshown in FIG. 7 is lower than the variation example.

The geometry and the material of the markers 217 and 218 are not limitedto the exemplified ones. For example, the marker 218 shown in FIG. 12shows the geometry in which one horizontal line portion 405 isorthogonal to one vertical line portion 406, but two or more horizontalline portions 405 or vertical line portions 406 can be applied. It isnot necessary to have the geometry of straight lines orthogonal to eachother, but any geometry such as a polygon, circle, oval, etc. can beapplied as the marker 218.

For example, a spherical marker with a hole to insert the handle portionof the driver 202 therein can be used instead of the radially symmetricmarker 217. The spherical marker improves a recognition rate in imagerecognition because a shape of the spherical marker in the capturedimage is constantly a circle, regardless of the capture angle of the CCDcameras 206 and 207.

For example, a predetermined pattern may be formed on a surface of thespherical marker and the pattern may be made by a retroreflectivematerial and a low reflectance material. The pattern enables the PC 209to calculate a position and a direction of the driver 202 with thespherical marker so that the PC 209 can calculate the coordinates ^(a)p₆of the point P₆ indicating the tip 225 of the driver 202.

In the example above, three markers 218 are attached to each of thesurfaces A through D of the operation table 203, but it is obviouslypossible to use one marker formed such that three cross-shapedprojections 402 are extended from the face 403 of one elongated sheetform base unit 401, and it is clear that such formed marker have thesame effects as the marker 218 shown in FIG. 12. In this case, the threecross-shaped projections 402 are arranged on the face 403 of theelongated sheet form base unit 401 at the interval as illustrated inFIG. 8, for example.

The markers 217 and 218 can be made of different materials havinggreatly different reflectances so that they can be easily detected inthe image recognition. Therefore, the marker 218 can have a lowreflectance sheet form base unit 401 and a high reflectance cross-shapedprojection 402. Similarly, the marker 217 can have a low reflectancecylindrical base unit 408 and high reflectance cylindrical projections409 a through 409 d.

The markers 217 and 218 shown in FIGS. 12 through 13B have the meritsthat they do not require electric power, and can be attached to anexisting driver 202 and operation table 203 without changing them.However, some modes for embodying the present invention allow ahigh-brightness light emission element such as an LED (light emittingdiode) etc. or an ultrasonic generator to be used as the markers 217and/or 218. That is, since it is only necessary to detect the positionin the three-dimensional space, the mode for embodying the presentinvention shown in FIG. 4 can be changed to use a sensor such as anultrasonic sensor, a vibration sensor, etc. for use in athree-dimensional measure.

For example, when an LED is used for the markers 217 and/or 218, themarkers can be recognized with high accuracy by using the CCD cameras206 and 207 of the wavelength sensitivity for the wavelength of thelight emitted by the LED. When an ultrasonic generator is used as themarkers 217 and/or 218, an ultrasonic signal is received by a pluralityof ultrasonic receivers whose positional relations are mutually knownand which replace the CCD cameras 206 and 207, thereby detecting thepositions of the markers from the difference in propagation time of theultrasonic signal.

When an LED or an ultrasonic generator is used as the marker 217, thecompletion signal generation tool 110 and the marker 217 can be combinedas one device. For example, when the torque applied to a screw by thedriver 202 reaches a specified torque, the ultrasonic generator attachedto the driver 202 generates an ultrasonic signal as a completion signal,a plurality of ultrasonic receivers receive the ultrasonic signal, andthe PC 209 can calculate the position of the ultrasonic generator.

However, when the LED and/or the ultrasonic generator are used, therearise the problem that the driving power such as electric power etc. isconsumed and the problem that a battery or wiring is necessary forsupply of the driving power. It may be difficult to newly provide thewiring for the existing driver 202 or operation table 203. It may alsobe difficult to provide the wiring without twisting or cutting thewiring also without varying the movable area of the operation table 203.

In the above-mentioned modes for embodying the present invention, twoCCD cameras 206 and 207 are used, but three or more CCD cameras can beused to eliminate a blind spot etc.

The CCD cameras 206 and 207 can capture a luminance image, that is, amonochrome gray-scale image, or a color image.

In the former case, there are the merits that a smaller amount of datais required for image processing, no adjustment to a color filter isrequired, etc. Furthermore, since the markers 217 and 218 shown in FIGS.12 through 13B are made of a high reflectance material and a lowreflectance material, they can be easily recognized in a luminanceimage.

On the other hand, when the CCD cameras 206 and 207 for capturing acolor image are used, for example, the markers 218 having redcross-shaped projections 402 can be attached to the surface A, and themarkers 218 having blue cross-shaped projections 402 can be attached tothe surface C. Thus, a color difference can be used in determining thedirection of the operation table 203. Thus, it is also possible tomoderate the restrictions on the attachment position of the markers 218on the surfaces A through D as shown in FIG. 9. That is, the practicalconfiguration and arrangement of the markers 218 are optionallydetermined only if the surfaces of the operation table 203 can bediscriminated from the appearance including the shape, color,arrangement pattern, etc. of the markers 218.

The contents of the operation in the above-mentioned modes for embodyingthe present invention relate to a screwing operation. However, othermodes of embodiments can be directed to other type of operations. Forexample, some modes of embodiments can be directed to a manufacturingprocess including the operations of soldering, welding, seizing,caulking, swaging, spot welding, bonding, etc. In addition, someembodiments can be applied to the case where one process includes pluraltypes of operations such as screwing, soldering, etc.

For example, in the process of ultrasonic welding for bonding resinplates at plural points, an ultrasonic welding unit issues a completionsignal each time a welding operation is completed at each welding point,and the PC 209 receives the completion signal through the IO board 215.Thus, if a tool configured to issue a completion signal when anoperation is completed is used, the above-described embodiments can bemodified to be applied to operations other than a screwing operation.

Some other embodiments can be applied to a process other than anassembly process. For example, an inspecting process etc. other than theassembly process can be included in the manufacturing process. In theinspecting process, an inspection can be made using a tool. If aconfiguration is designed such that an inspecting tool can issue acompletion signal each time one inspecting operation is completed, thenan embodiment can be applied to the inspecting process.

The tool itself can be hand-powered, not requiring electric power. Forexample, if the completion of an operation can be detected by avibration sensor, the vibration sensor is available in an embodiment sothat the vibration sensor is attached to the hand-powered tool andissues a completion signal.

The operation table 203 shown in FIG. 4 is designed for enablingposition movement, rotation of a table plate on the horizontal axis, androtation of a table plate on the vertical axis. However, it is obviousthat an operation table of another type can also be used such as anoperation table whose table plate is designed not to rotate on thehorizontal axis. In this case, the reduction of the freedom in themovement of the operation table enables the moderation of therestrictions in the arrangement pattern of the markers 218.

Although the operation position of a screwing operation is a point, butthere is an operation to cover a range as a linear or planar extension.For example, when metal plates are processed by fillet weld, the weldingis linearly performed. Thus, some embodiments can be applied to anoperation performed on a range having a linear or planar expansion.

In an operation performed on a range having a linear or planarexpansion, for example, a tool configured to intermittently issue asignal at predetermined intervals during the operation is used. Thesignal is called an “operation in-progress signal”. Like the completionsignal, an operation in-progress signal is received by the PC 209through the driver controller 210 and the IO board 215. The position inwhich an operation is performed when the PC 209 receives the operationin-progress signal is called an “operation in-progress position”.

The operation position is represented as a set of coordinates in theworkpiece coordinate system Σw of the positions of points existing andextending linearly or as a plane, or represented by an equationexpressing a line or a plane in the workpiece coordinate system Σw. Uponreceipt of the operation in-progress signal, the PC 209 performs aprocess similar to one performed when the completion signal is received.That is, the PC 209 detects the position and direction of the operationtable 203 when receiving an operation in-progress signal, calculates aparameter for transformation from the absolute coordinate system La tothe workpiece coordinate system Σw, detects the coordinates of anoperation in-progress position in the absolute coordinate system Σa,transforms the detected coordinates of the operation in-progressposition in the absolute coordinate system Σa into the workpiececoordinate system Σw, compares the transform result with the operationcoordinates stored in advance, and locates the operation positioncorresponding to the operation in-progress position.

In this case, an operation performed linearly or on a plane correspondsto a locus of points including one or more operation in-progresspositions with which respective points of the operation position havebeen located and one operation completion position. Some embodiments arethus applied to the operation performed linearly or on a plane. Thetiming of issuing an operation in-progress signal is optionallydetermined.

In the above-mentioned mode for embodying the present invention, bothabsolute coordinate system Σa and workpiece coordinate system Σw areorthogonal coordinate systems, but it is obvious that other coordinatesystems such as a polar coordinate system etc. can be used.

In the example shown in FIG. 11, there are two areas graphicallyrepresenting the workpiece 204. On the other hand, there is theworkpiece 204 only requiring an operation on one surface, and theworkpiece 204 requiring an operation on three or more surfaces. It ispreferable to appropriately adjust the number of display areas dependingon the number of surfaces to be processed. When there are a plurality ofoperations to be performed in one process, and the operations are to beperformed in a predetermined order, it is preferable that the operationmonitor 212 displays the instruction to indicate the order of theoperations.

1. An operation position locating system which locates a position on aproduct where an operation is performed in a manufacturing process ofthe product including one or more operations to be performed using atool, comprising: an operation completion coordinates detection unitprocessing a plurality of images obtained by capturing from a pluralityof viewpoints a range including an operation completion position as aposition on which an operation is completed when the operation iscompleted, thereby detecting a set of operation completion coordinatesrepresenting the operation completion position; and an operationposition locating unit obtaining a set of operation coordinatesrepresenting an operation position as a position on which the operationis to be performed on each of the one or more operations, and locatingthe operation position corresponding to the operation completionposition based on the set of operation completion coordinates and theone or more obtained sets of operation coordinates.
 2. The operationposition locating system according to claim 1, wherein the operationcompletion coordinates detection unit detects a position and a directionof the tool, and calculates the set of operation completion coordinatesbased on the detected position and direction of the tool.
 3. Theoperation position locating system according to claim 2, wherein theposition and the direction of the tool are detected by detecting a toolmarker attached to the tool by image recognition.
 4. The operationposition locating system according to claim 3, wherein the tool markercomprises a base unit and a plurality of projections extended from thebase unit; the base unit has high reflectance and the plurality ofprojections have low reflectance, or the base unit has low reflectanceand the plurality of projections have high reflectance; and each of theplurality of the projections takes a form of a substantially radialsymmetry about an axis common among the plurality of projections whenthe tool marker is attached to the tool.
 5. The operation positionlocating system according to claim 1, wherein the operation completioncoordinates are represented by a first coordinate system; the operationcoordinates are represented by a second coordinate system based on theproduct; the operation position locating system further comprises aparameter determination unit detecting a position and directionrepresented by the first coordinate system of the product or an objecthaving a predetermined positional relation with the product, therebydetermining a parameter value required to coordinate transformationbetween the first coordinate system and the second coordinate system;and the operation position locating unit transforms coordinates usingthe parameter to represent the operation completion position and theoperation position by a same coordinate system, and locates theoperation position corresponding to the operation completion position.6. The operation position locating system according to claim 5, whereinthe coordinate transformation is a transformation from the firstcoordinate system to the second coordinate system.
 7. The operationposition locating system according to claim 5, wherein: the operation isperformed with the product fixed to an operation table at least one ofwhose position and direction can be adjusted, and which is provided withan operation table marker; the range captured in the plurality of imagesincludes the operation table; the parameter determination unit detects aposition and direction of the operation table represented by the firstcoordinate system by detecting the operation table marker on theplurality of images by image recognition, and determines the parametervalue based on the position and direction of the operation table.
 8. Theoperation position locating system according to claim 7, wherein: theoperation table marker has a base unit having an attachment surface tothe operation table and a projection extending from the base unit; thebase unit has high reflectance and the projection has low reflectance,or the base unit has low reflectance and the projection has highreflectance; a plurality of projections correspond to the operationtable as a result of attaching a plurality of the operation table markerto the operation table, or as a result of providing a plurality ofprojections to the operation table marker; and the position and thedirection of the operation table are detected based on at least one of adifference in appearance of the plurality of projections, and anarrangement pattern of the plurality of projections.
 9. The operationposition locating system according to claim 1, further comprising adisplay unit displaying operation information about the operationcorresponding to the located operation position by using a character, asymbol, an image, sound, or a combination thereof.
 10. The operationposition locating system according to claim 1, wherein processinformation indicating which operation of the operations to be performedhas been completed is managed based on a result of locating theoperation position.
 11. The operation position locating system accordingto claim 1, wherein: the operation is a screwing operation; the tool isa torque driver capable of detecting that a specified torque has beenreached; the torque driver outputs a completion signal when a torqueapplied to the screw has reached the specified torque; the operationcompletion coordinates detection unit receives the completion signal anddetects the set of operation completion coordinates upon receipt of thecompletion signal.
 12. The operation position locating system accordingto claim 1, wherein: the operation is to be performed on a range havinglinear or planar expansion; the operation completion coordinatesdetection unit receives an operation in-progress signal output one ormore times during progress of the operation, and detects a set ofoperation in-progress coordinates indicating a position in which theoperation is being performed when the operation in-progress signal isreceived; and the operation position locating unit further locates theoperation position corresponding to the set of operation in-progresscoordinates.
 13. An operation position locating system which locates aposition on a product where an operation is performed in a manufacturingprocess of the product including one or more operations to be performedusing a tool, comprising: an operation completion coordinates detectionunit detecting a set of operation completion coordinates in which anoperation completion position as a position where the operation iscompleted is represented by a first coordinate system; a parameterdetermination unit detecting a position and direction represented by thefirst coordinate system of the product or an object having apredetermined positional relation with the product, thereby determininga parameter value required to coordinate transformation between thefirst coordinate system and a second coordinate system based on theproduct; an operation position locating unit obtaining a set ofoperation coordinates in which an operation position as a position wherethe operation is to be performed is represented by the second coordinatesystem on each of the one or more operations, transforming coordinatesusing the parameter to represent the operation completion position andthe operation position by a same coordinate system, and locating theoperation position corresponding to the operation completion position.14. The operation position locating system according to claim 13,wherein the operation completion coordinates detection unit detects theoperation completion coordinates using output of a sensor attached tothe tool and capable of performing a three-dimensional measurement. 15.The operation position locating system according to claim 13, whereinthe parameter determination unit detects the position and the directionof the product or the object represented in the first coordinate systemusing a sensor capable of performing a three-dimensional measurement.16. An operation cell corresponding to a process including one or moreoperations to be performed using a tool in a manufacturing process of aproduct including one or more processes, comprising: an operationcompletion coordinates detection unit processing a plurality of imagesobtained by capturing from a plurality of viewpoints a range includingan operation completion position as a position on which an operation iscompleted when the operation is completed, thereby detecting a set ofoperation completion coordinates representing the operation completionposition; and an operation position locating unit obtaining a set ofoperation coordinates representing an operation position as a positionon which the operation is to be performed on each of the one or moreoperations, and locating the operation position corresponding to theoperation completion position based on the set of operation completioncoordinates and the one or more obtained sets of operation coordinates.17. A method performed in a manufacturing process of a product, themanufacturing process including one or more operations to be performedusing a tool, comprising: processing a plurality of images obtained bycapturing from a plurality of viewpoints a range including an operationcompletion position as a position on which an operation is completedwhen the operation is completed; detecting a set of operation completioncoordinates representing the operation completion position based on aresult of processing the plurality of images; obtaining a set ofoperation coordinates representing an operation position as a positionon which the operation is to be performed on each of the one or moreoperations; and locating the operation position corresponding to theoperation completion position based on the set of operation completioncoordinates and the one or more obtained sets of operation coordinates.18. The method according to claim 17, wherein the product ismanufactured in the manufacturing process while locating each operationposition for each operation.
 19. A computer-readable storage mediumstoring a program used to direct a computer to locate a position on aproduct where an operation is performed in a manufacturing process ofthe product including one or more operations to be performed using atool, comprising: a step of processing a plurality of images obtained bycapturing from a plurality of viewpoints a range including an operationcompletion position as a position on which an operation is completedwhen the operation is completed, thereby detecting a set of operationcompletion coordinates representing the operation completion position; astep of obtaining a set of operation coordinates representing anoperation position as a position on which the operation is to beperformed on each of the one or more operations; and a step of locatingthe operation position corresponding to the operation completionposition based on the set of operation completion coordinates and theone or more obtained sets of operation coordinates.